Session 5: linear regression

Load the dataset

The collected dataset includes information about houses on sale in the Dublin area. Each house is an entry of the dataset: a mixed-type data comprising of numerical, categorical and textual data.

The goal is to combine both numerical/categorical features and textual features to predict if the house-price is above or below 550,000

The house price is determined by some factors like

• location (area),
• surface (size),
• the number of bedrooms,
• the number of bathrooms,
• property type,
• house-features (size of the windows, construction material).

The physical attributes of the house such as the number of bedrooms, the number of bathrooms, the surface of the house, property type, and its location are information that is directly accessible from the dataset. Instead, the house-features can (sometimes only indirectly) be inferred from the house-description, house-facility and house-features. You can download the dataset from this url: https://github.com/benavoli/ST8003/tree/main/session05 You can see a typical entry in the dataset hereafter

data <- read.csv(file = '../session5/train.csv',sep="," )
data['pricerange']<-as.vector(data['price']>550000)+0.0# we make a column which is 1 when price 
#is above 550000 and zero otherwise
data[1,]

Data Cleaning, Covariate selection and preprocessing

We select some of the columns (‘bathrooms’,‘beds’,‘surface’) we will use as predictors for price

datasel = data[c('bathrooms','beds','surface','pricerange')]
datasel = na.omit(datasel)# we remove all the rows including nan
datasel

Linear regression

We now fit linear regression

model = glm(pricerange ~ bathrooms + beds + surface,  family = "binomial", data = datasel)

glm.fit: fitted probabilities numerically 0 or 1 occurred

summary(model)

Call:
glm(formula = pricerange ~ bathrooms + beds + surface, family = "binomial", 
    data = datasel)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-3.2076  -0.6062  -0.3426   0.5130   3.3574  

Coefficients:
              Estimate Std. Error z value Pr(>|z|)    
(Intercept) -5.646e+00  2.300e-01 -24.544  < 2e-16 ***
bathrooms    3.693e-01  6.577e-02   5.615 1.97e-08 ***
beds         1.223e+00  7.267e-02  16.824  < 2e-16 ***
surface      6.822e-05  2.379e-05   2.867  0.00415 ** 
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 2889.9  on 2401  degrees of freedom
Residual deviance: 2022.6  on 2398  degrees of freedom
AIC: 2030.6

Number of Fisher Scoring iterations: 5

Is this a good model? Can we use other columns in data to improve the model? Can we include polynomial and/or interaction terms to improve the model? Use the model selection approaches you learned in session 5 and 6 to find a better model.

Unseen data

You can test the predictive performance of our best model on unseen data

datatest <- read.csv(file = '../session5/test.csv',sep="," )
datatest[10:28,3:16]#one of the entries, there are 16 columns, the first two columns are just ids. The price column is not reported. You have to predict the price for all the entries in dataset

Prediction

predictions <- predict(model,datatest, type="response")
predictions[1:5]

        1         2         3         4         5 
0.4980193 0.2257686 0.5898662 0.3792981 0.1675038 

these are the predicted probabilities for pricerange to be 1 for 5 houses in the dataset. You can save and submit your best predictions for our internal data science competition. This is the code, which uses the threshold 0.5 to predict 1, that is house price above 550000.

write.csv(predictions,"name_surname.csv")

We will use accuracy score to evaluate the accuracy of your predictions.